(Some adjacent or related issues #22790 #22946 #15840 #21518 so I don't need to search for them again.)

@tobni As I understand it there isn't a stable ABI for free threaded until 3.15. Having dug in to do this work, do you have a sense of how much of a problem or risk that is (or is not)?

@cburroughs Risk is close to none afaik, it is mostly a pyo3 compat (meaning we couldnt build it) issue. If we are allowed to assume pants is ran as a scie we defined, the internal 3rd party lockfile does the rest. I think that is ok, building a pants dist and targeting a different python would be highly special usecase.

If a plugin uses native deps in rule code that could bite them iff there is no 3.14t wheels for them.

Pretty substantial (and consequential) change - so I tacked on a bunch of reviewers

Once #23482 is in this will be rebased and the pex binary will again be stripped.

Edit: Done.

General review comments:

1. This PR should probably be split into distinct components to make review easier. For example, the LockMap change and the related memoization changes can be a standalone PR.

2. Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

We could vendor release versions with t-suffixes to make it target free-threaded? I'm doubtful of the value to be honest, but that'd be simple enough implementation wise (but requires CI/CD work).

I'd prefer opt-out though. I dont want to treat this as a scary change when it isnt, but I will also not claim 0 impact across all plugin code.

For the first comment: Yes, easily split. Each commit ~stands on its own already so I can do that branch-ectomy with ~0 effort.

We could vendor release versions with t-suffixes to make it target free-threaded? I'm doubtful of the value to be honest, but that'd be simple enough implementation wise (but requires CI/CD work).

So like everything in software engineering, my view is "it depends." My preference here is that we not introduce a one way version gate on users when there is no one dedicated to fixing issues for them in a timely manner. I'd rather avoid users being potentially stuck on 2.32 or lower versions with no way to consume all of the other fixes Pants has in 2.33 unrelated to free-threading.

So how much more complex would it be to release a free-threading version of Pants alongside the GIL version? The main artifact is the engine's shared library (native_engine.so), can we just build it twice? (once with 3.14 GIL and once with 3.14 free-threaded)?

My view really depends on just how much additional work the CI/CD & release changes would be.

I'd prefer opt-out though. I dont want to treat this as a scary change when it isnt, but I will also not claim 0 impact across all plugin code.

Opt-out is fine. The concern for me is having the ability to control adoption of the change with some sort of flag versus forcing users to stay on a prior versions because there is no escape hatch from this change. I submit that Incremental adoption with an escape hatch makes for a better user experience.

So like everything in software engineering, my view is "it depends." My preference here is that we not introduce a one way version gate on users when there is no one dedicated to fixing issues for them in a timely manner. I'd rather avoid users being potentially stuck on 2.32 or lower versions with no way to consume all of the other fixes Pants has in 2.33 unrelated to free-threading.

You used specific version numbers so I'll just clarify that I do not think this is a 2.33.x change but rather a 2.34.x feature. The point is of course still valid. I'm not sure I agree that this is a strong case for pants and free threading though, we have a patch-version series for fixes and we vendor a binary. We can choose to only release free threading and address eventual bugs. Because it will be bugs, users being "stuck" will be on some flaky CI-runs surfacing races we missed, most likely. Unless of course there is ABI-incompatible user plugins out there.

So how much more complex would it be to release a free-threading version of Pants alongside the GIL version? The main artifact is the engine's shared library (native_engine.so), can we just build it twice? (once with 3.14 GIL and once with 3.14 free-threaded)?

I dont know exactly, the main effort is figuring that out, and is the counterweight in terms of value I was refering to.

Now I know some more - It'd have to be a separate pants.toml field to not mess with PEP440 compat, but otherwise it goes:

1. One release, two asset sets. The GitHub release has to publish both scies per platform eg: pants.2.34.0-linux-x86_64.scie and a free threaded sibling whose naming is TBD.
2. Selection is a scie-pants config knob. scie-pants reads from pants.toml. scie-pants maps the pants.toml field to the asset-name suffix and fetches.

Essentially a question of if we're okay to publish 2x scies.

General review comments:

1. This PR should probably be split into distinct components to make review easier. For example, the LockMap change and the related memoization changes can be a standalone PR.
2. Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

@tdyas 1 is now done. See the list in the pull request description.

I don't think publishing multiple scies is a problem. But historically changing the minimum scie-pants version has been perceived as disruptive to users and we have tried not to do so "often". (Making it not so disruptive is the impetuous for #22946) So however we decide on opt in/out, I think it is important than users on old scie-pants versions get something that works (that is, one of the scies follows the current naming scheme).

https://www.pantsbuild.org/dev/docs/contributions/releases/release-strategy says "every six weeks", but that isn't set in stone and we often go longer. We can also choose to go shorter if that helps manage risk, for example by having a "everything but flipping the switch release" followed shortly by a "just flips the switch" one.

Thanks for breaking this apart into smaller chunks and I'm excited to see this move forward.

I just want to point out that not everyone uses a scie. We use the big monolithic binaries that are shipped in the release. So please make sure to build an ft version of that as well.

I just want to point out that not everyone uses a scie. We use the big monolithic binaries that are shipped in the release. So please make sure to build an ft version of that as well.

To make sure I have this right, what from https://github.com/pantsbuild/pants/releases/tag/release_2.32.1 do you use? Colloquially If I head "big monolithic binaries" I would think of the scies there (ex: pants.2.32.1-cp314-linux_x86_64), while scie-pants today uses the .pex files (ex : pants.2.32.1-cp314-linux_x86_64.pex ).

We actually don't need to ship multiple scies/versions I think, the free-threaded Python builds support re-enabling the GIL with the PYTHON_GIL environment variable https://docs.python.org/3/howto/free-threading-python.html#the-global-interpreter-lock-in-free-threaded-python

We actually don't need to ship multiple scies/versions I think, the free-threaded Python builds support re-enabling the GIL with the PYTHON_GIL environment variable https://docs.python.org/3/howto/free-threading-python.html#the-global-interpreter-lock-in-free-threaded-python

That doesnt cover ABI mismatch in user-written plugins third party deps, which is imo the strongest argument for a non free-threaded variant.

That said, PYTHON_GIL does work and will launch the pants daemon with GIL enabled under free-threaded. It's a bit fickle though, the daemon doesnt pick it up mid run.

To make sure I have this right, what from https://github.com/pantsbuild/pants/releases/tag/release_2.32.1 do you use? Colloquially If I head "big monolithic binaries" I would think of the scies there (ex: pants.2.32.1-cp314-linux_x86_64), while scie-pants today uses the .pex files (ex : pants.2.32.1-cp314-linux_x86_64.pex ).

Yes, sorry. I mis-spoke. We don't use scie-pants.